2 links tagged with all of: performance + query optimization
Click any tag below to further narrow down your results
Links
A developer measured five common SQL patterns that force PostgreSQL to do unnecessary work—UNION instead of UNION ALL, MATERIALIZED CTEs, ORDER BY random(), SELECT * instead of specific columns, and unindexed expressions—and found gaps ranging from 20x to 4,000x slower on a 2 million row table.
- UNION deduplicated 2 million rows unnecessarily (13.6s) versus UNION ALL (0.68s) when the developer knew the halves couldn't overlap
- MATERIALIZED keyword forced PostgreSQL to build the entire CTE before filtering, turning a 0.24ms query into 1,973ms
- ORDER BY random() LIMIT 1 sorted all 2 million rows to get one (1,313ms); TABLESAMPLE SYSTEM did it in 0.3ms, though with different guarantees
- SELECT * required visiting the table for every row (1,718 buffers, 64.8ms) versus querying just the indexed column (279 buffers, 32.8ms)
High CPU utilization doesn't mean your Spark jobs are efficient — it just means they're busy. This article walks through four layers where waste hides: misaligned executor configs, skewed data distribution, slow storage access, and wasteful application logic that processes data you later discard.
- Executor shape matters as much as utilization: a poorly sized executor can max out CPU while leaving machine capacity stranded. Fixing core-to-memory ratios cut one workload's vCore-hours by 54% and runtime by 83%.
- Data skew concentrates work into a few slow tasks while others finish early. Identifying the specific hot key (market_id=2147) and salting it across partitions reduced one stage's runtime by 24%.
- Spark often spends time listing and reading data it doesn't need before it even starts processing. Late filtering in query plans can keep a cluster busy processing billions of intermediate records that get discarded later.